Detecting user’s eye movement using sensors in hearing instruments

ABSTRACT

A set of one or more processing circuits obtains eye movement-related eardrum oscillation (EMREO)-related measurements from one or more EMREO sensors of a hearing instrument. The EMREO sensors are located in an ear canal of a user of the hearing instrument and are configured to detect environmental signals of EMREOs of an eardrum of the user of the hearing instrument. The one or more processing circuits may perform an action based on the EMREO-related measurements.

This application is a continuation of U.S. application Ser. No.17/568,292, filed Jan. 4, 2022, which is a continuation of U.S.application Ser. No. 16/799,390, filed Feb. 24, 2020, now issued as U.S.Pat. No. 11,223,915, which claims the benefit of U.S. Provisional PatentApplication 62/832,598, filed Apr. 11, 2019, and U.S. Provisional PatentApplication 62/810,298, filed Feb. 25, 2019, the entire content of eachof which is hereby incorporated by reference.

TECHNICAL FIELD

This disclosure relates to hearing instruments.

BACKGROUND

Hearing instruments are devices designed to be worn on, in, or near oneor more of a user's ears. Common types of hearing instruments includehearing assistance devices (e.g., “hearing aids”), earbuds, headphones,hearables, cochlear implants, and so on. In some examples, a hearinginstrument may be implanted or osseointegrated into a user. Some hearinginstruments include additional features beyond just environmentalsound-amplification. For example, some modern hearing instrumentsinclude advanced audio processing for improved device functionality,controlling and programming the devices, and beamforming, and some caneven communicate wirelessly with external devices including otherhearing instruments (e.g., for streaming media).

SUMMARY

This disclosure describes techniques for detecting and performingactions based on eye movement-related eardrum oscillations (EMREOs) ofone or more eardrums of a user of one or more hearing instruments. Forinstance, a set of one or more processing circuits may obtainEMREO-related measurements from one or more EMREO sensors of a hearinginstrument. The EMREO sensors are located in an ear canal of a user ofthe hearing instrument and are configured to detect environmentalsignals of EMREOs of an eardrum of the user of the hearing instrument.The one or more processing circuits may perform an action based on theEMREO-related measurements.

In one aspect, this disclosure describes a method comprising: obtaining,by a set of one or more processing circuits, eye movement-relatedeardrum oscillation (EMREO)-related measurements from one or more EMREOsensors of a hearing instrument, wherein the EMREO sensors are locatedin an ear canal of a user of the hearing instrument and are configuredto detect environmental signals of EMREOs of an eardrum of the user ofthe hearing instrument and generate the EMREO-related measurements basedon the detected environmental signals; and performing, by the one ormore processing circuits, an action based on the EMREO-relatedmeasurements.

In another aspect, this disclosure describes a system comprising: one ormore EMREO sensors located in an ear canal of a user of a hearinginstrument, wherein the EMREO sensors are configured to detectenvironmental signals of EMREOs of an eardrum of the user and generatethe EMREO-related measurements based on the detected environmentalsignals; and one or more processing circuits configured to: obtainEMREO-related measurements from the one or more EMREO sensors; andperform an action based on the EMREO-related measurements.

In another aspect, this disclosure describes a system comprising: meansfor obtaining EMREO-related measurements from one or more EMREO sensorsof a hearing instrument, wherein the EMREO sensors are located in an earcanal of a user of the hearing instrument and are configured to detectenvironmental signals of EMREOs of an eardrum of the user of the hearinginstrument and generate the EMREO-related measurements based on thedetected environmental signals; and means for performing an action basedon the EMREO-related measurements.

In another aspect, this disclosure describes a computer-readable storagemedium having instructions stored thereon that, when executed, cause oneor more processing circuits to: obtain eye movement-related eardrumoscillation (EMREO)-related measurements from one or more EMREO sensorsof a hearing instrument, wherein the EMREO sensors are located in an earcanal of a user of the hearing instrument and are configured to detectenvironmental signals of EMREOs of an eardrum of the user of the hearinginstrument and generate the EMREO-related measurements based on thedetected environmental signals; and perform an action based on theEMREO-related measurements.

The details of one or more aspects of the disclosure are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the techniques described in this disclosurewill be apparent from the description, drawings, and claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a conceptual diagram illustrating an example system thatincludes one or more hearing instrument(s), in accordance with one ormore techniques of this disclosure.

FIG. 2 is a conceptual diagram illustrating example positions of hearinginstruments in the ear canals of a user, in accordance with one or moretechniques of this disclosure.

FIG. 3 is a conceptual diagram illustrating example hearing instrumentshaving eye movement-related eardrum oscillation (EMREO) sensors andelectrooculography (EOG) electrodes, in accordance with one or moretechniques of this disclosure.

FIG. 4 is a block diagram illustrating example components of a hearinginstrument, in accordance with one or more aspects of this disclosure.

FIG. 5 is a block diagram illustrating example components of a computingdevice, in accordance with one or more aspects of this disclosure.

FIG. 6 is a flowchart illustrating an example operation in accordancewith one or more example techniques described in this disclosure.

DETAILED DESCRIPTION

Recent research has demonstrated that eye movements trigger eardrumoscillations. For example, Gruters et al., “The eardrums move when theeyes move: A multisensory effect on the mechanics of hearing,” Proc.Natl. Acad. Sci., p. E1309-E1318, 2018, describes this phenomenon as eyemovement-related eardrum oscillations (EMREOs). These oscillationsproduce sinusoidal sound pressure levels in the ear canal ofapproximately 55dBA at 35 Hz. Such signals can be measured usingotoacoustic emissions (OAEs) testing equipment. Otoacoustic emissionsare soundwaves generated within the inner ear that propagate in alateral (outward) direction through an ear canal.

When the user's eyes move to the left, both eardrums move in sync to theright and then oscillate. When the eyes move to the right, both eardrumsmove in sync to the left and then oscillate. Eardrum vibration beginsapproximately 10 milliseconds (ms) before the eyes move and the eardrumvibration continues until shortly after the eyes reach fixation. Inother words, EMREOs began as early as 10 ms before saccade onset andlasted throughout the saccade and well into subsequent periods offixation. The total period of an EMREO is about 110 ms. Significantly,the phase (i.e., direction) of the initial movement of eardrums duringan EMREO corresponds to the direction of the eye movement in ahorizontal plane. In this context, the horizontal plane is a traverseplane that passes through both of the eyes of the user.

Another aspect of EMREOs is that the amplitude of an EMREO correspondsto the distance the eye moved relative to a fixation point at zerodegrees in the horizontal plane. Additionally, EMREOs are detectableeven in the presence of sound stimuli (clicks) that are commonly used toelicit OAEs. Moreover, EMREOs occurred in the absence or presence ofexternal sound stimuli but were not triggered nor affected by thosesounds. Furthermore, the amplitude of the EMREO was correlated stronglywith the distance of the eye movement, and the phase of the EMREO wascorrelated strongly with the direction of the eye movement. The phase ofthe EMREO refers to the position of a sound wave relative to a fixedpoint in time, such as saccade onset. In the context of this disclosure,phase is used to determine whether the pressure change in the ear canalis initially positive or negative in response to an eye movement. Thus,the phase of the EMREO may be considered to be the relative direction ofmovement of the eardrums at the onset of an EMREO. The frequency ofEMREOs is in the 20-40 Hz range.

Currently in a laboratory setting, eye movements are tracked usingadvanced sensors (e.g., cameras facing the eyes), or electrodes placedin or around the eyes and/or ears that require wired connections andsophisticated amplifiers. Furthermore, although EMREO signals have beenmeasured in a laboratory setting, challenges exist in detecting EMREOsusing a microphone in an in-situ hearing instrument due to environmentalfactors such as user motion, ambient noise, user speech, and otherfactors.

As described in this disclosure, example hearing instruments may includeone or more sensors configured to detect environmental signals ofEMREOs, and therefore eye movements. For example, a set of one or moreprocessors may obtain EMREO-related measurements from one or more EMREOsensors of one or more hearing instruments. The EMREO sensors arelocated in an ear canal of a user of the hearing instrument and areconfigured to detect environmental signals of EMREOs of an eardrum ofthe user of the hearing instrument. In this example, the one or moreprocessors may perform an action based on the EMREO-relatedmeasurements. For instance, the actions may include changing settingsfor the hearing instruments, generating information about the sleep ofthe user, controlling user interfaces, sending interpersonal messages,determining salient objects in the environment of the user, detectingepileptic seizures, and so on. This disclosure also proposessignal-processing and machine-learning techniques that could be used toenhance the measurement of eardrum vibrations and the translation to eyemovement information. Furthermore, this disclosure describes techniquesthat use information about the user's EMREOs to generate health-relateddata regarding the user. The health-related data may include datarelated to the user's physical health, mental or emotional health, or acombination thereof.

FIG. 1 is a conceptual diagram illustrating an example system 100 thatincludes hearing instruments 102A, 102B, in accordance with one or moretechniques of this disclosure. This disclosure may refer to hearinginstruments 102A and 102B collectively, as “hearing instruments 102.” Auser 104 may wear hearing instruments 102. In some instances, such aswhen user 104 has unilateral hearing loss, user 104 may wear a singlehearing instrument. In other instances, such as when user 104 hasbilateral hearing loss, the user may wear two hearing instruments, withone hearing instrument for each ear of the user.

Hearing instruments 102 may comprise one or more of various types ofdevices that are configured to provide auditory stimuli to a user andthat are designed for wear and/or implantation at, on, or near an ear ofthe user. Hearing instruments 102 may be worn, at least partially, inthe ear canal or concha. One or more of hearing instruments 102 mayinclude behind the ear (BTE) components that are worn behind the ears ofuser 104. In some examples, hearing instruments 102 comprise devicesthat are at least partially implanted into or osseointegrated with theskull of the user. In some examples, one or more of hearing instruments102 is able to provide auditory stimuli to user 104 via a boneconduction pathway.

In any of the examples of this disclosure, each of hearing instruments102 may comprise a hearing assistance device. Hearing assistance devicesinclude devices that help a user hear sounds in the user's environment.Example types of hearing assistance devices may include hearing aiddevices, Personal Sound Amplification Products (PSAPs), cochlear implantsystems (which may include cochlear implant magnets, cochlear implanttransducers, and cochlear implant processors), and so on. In someexamples, hearing instruments 102 are over-the-counter,direct-to-consumer, or prescription devices. Furthermore, in someexamples, hearing instruments 102 include devices that provide auditorystimuli to the user that correspond to artificial sounds or sounds thatare not naturally in the user's environment, such as recorded music,computer-generated sounds, or other types of sounds. For instance,hearing instruments 102 may include so-called “hearables,” earbuds,earphones, or other types of devices. Some types of hearing instrumentsprovide auditory stimuli to the user corresponding to sounds from theuser's environmental and also artificial sounds.

In some examples, one or more of hearing instruments 102 includes ahousing or shell that is designed to be worn in the ear for bothaesthetic and functional reasons and encloses the electronic componentsof the hearing instrument. Such hearing instruments may be referred toas in-the-ear (ITE), in-the-canal (ITC), completely-in-the-canal (CIC),or invisible-in-the-canal (IIC) devices. In some examples, one or moreof hearing instruments 102 may be behind-the-ear (BTE) devices, whichinclude a housing worn behind the ear contains all of the electroniccomponents of the hearing instrument, including the receiver (i.e., thespeaker). The receiver conducts sound to an earbud inside the ear via anaudio tube. In such examples, the earbud may include EMREO sensors 106Aor 106B. In some examples, one or more of hearing instruments 102 may bereceiver-in-canal (RIC) hearing-assistance devices, which include ahousing worn behind the ear that contains electronic components and ahousing worn in the ear canal that contains the receiver.

Hearing instruments 102 may implement a variety of features that helpuser 104 hear better. For example, hearing instruments 102 may amplifythe intensity of incoming sound, amplify the intensity of certainfrequencies of the incoming sound, or translate or compress frequenciesof the incoming sound. In another example, hearing instruments 102 mayimplement a directional processing mode in which hearing instruments 102selectively amplify sound originating from a particular direction (e.g.,to the front of the user) while potentially fully or partially cancelingsound originating from other directions. In other words, a directionalprocessing mode may selectively attenuate off-axis unwanted sounds. Thedirectional processing mode may help users understand conversationsoccurring in crowds or other noisy environments. In some examples,hearing instruments 102 may use beamforming or directional processingcues to implement or augment directional processing modes. Exampletechniques for beamforming are described in U.S. Pat. No. 10,425,745,issued Sep. 24, 2019. Example directional processing cues may includepinna cues, interaural time and level differences, and so on. In someexamples, hearing instruments 102 may communicate with an externalbeamforming device to obtain information about a preferred direction oflistening attention.

In some examples, hearing instruments 102 may reduce noise by cancelingout or attenuating certain frequencies. Furthermore, in some examples,hearing instruments 102 may help user 104 enjoy audio media, such asmusic or sound components of visual media, by outputting sound based onaudio data wirelessly transmitted to hearing instruments 102.

Hearing instruments 102 may be configured to communicate with eachother. For instance, in any of the examples of this disclosure, hearinginstruments 102 may communicate with each other using one or morewirelessly communication technologies. Example types of wirelesscommunication technology include Near-Field Magnetic Induction (NFMI)technology, a 900 MHz technology, a BLUETOOTH™ technology, a WI-FI™technology, audible sound signals, ultrasonic communication technology,infrared communication technology, an inductive communicationtechnology, or another type of communication that does not rely on wiresto transmit signals between devices. In some examples, hearinginstruments 102 use a 2.4 GHz frequency band for wireless communication.In examples of this disclosure, hearing instruments 102 may communicatewith each other via non-wireless communication links, such as via one ormore cables, direct electrical contacts, and so on.

In the example of FIG. 1 , hearing instrument 102A includes a set of oneor more eye movement-related eardrum oscillation (EMREO) sensors 106Aand hearing instrument 102B may include a set of EMREO sensors 106B.This disclosure may refer to EMREO sensors 106A and EMREO sensors 106Bcollectively as “EMREO sensors 106”. In other examples of thisdisclosure, only one of hearing instruments 102 includes EMREO sensors.EMREO sensors 106 may be located within the ear canals of user 104.EMREO sensors 106 may be configured to detect environmental signals ofEMREOs of user 104. The environmental signals of EMREOs are signalsproduced in the environment when EMREOs occur. Example environmentalsignals of EMREOs may include soundwaves caused by EMREOs, transmissionwaves through skin of the ear canals of user 104 caused by EMREOs,signals (e.g., structure light signals, reflected laser light, etc.)from which the position and/or shape of the eardrums may be determined,and so on.

As shown in the example of FIG. 1 , system 100 may also include acomputing system 108. In other examples, system 100 does not includecomputing system 108. Computing system 108 comprises one or morecomputing devices, each of which may include one or more processors. Forinstance, computing system 108 may comprise one or more mobile devices,server devices, personal computer devices, handheld devices, wirelessaccess points, smart speaker devices, smart televisions, medical alarmdevices, smart key fobs, smartwatches, smartphones, motion or presencesensor devices, smart displays, screen-enhanced smart speakers, wirelessrouters, wireless communication hubs, prosthetic devices, mobilitydevices, special-purpose devices, accessory devices, and/or other typesof devices. Accessory devices may include devices that are configuredspecifically for use with hearing instruments 102. Example types ofaccessory devices may include charging cases for hearing instruments102, storage cases for hearing instruments 102, media streamer devices,phone streamer devices, external microphone devices, remote controls forhearing instruments 102, and other types of devices specificallydesigned for use with hearing instruments 102. Actions described in thisdisclosure as being performed by computing system 108 may be performedby one or more of the computing devices of computing system 108. One ormore of hearing instruments 102 may communicate with computing system108 using wireless or non-wireless communication links. For instance,hearing instruments 102 may communicate with computing system 108 usingany of the example types of communication technologies describedelsewhere in this disclosure.

As described in this disclosure, a set of one or more processors (whichmay include processors in one or more of hearing instrument 102A,hearing instrument 102B, and computing system 108) may obtainEMREO-related measurements from one or more EMREO sensors 106 of one ormore of hearing instruments 102. As noted above, EMREO sensors 106 maybe located in one or more of the ear canals of user 104 and may beconfigured to detect environmental signals of EMREOs of one or more ofthe eardrums of user 104. The processors may perform one or more actionsbased on the EMREO-related measurements.

FIG. 2 is a conceptual diagram illustrating example positions of hearinginstruments 102 in the ear canals 200A, 200B of user 104, in accordancewith one or more techniques of this disclosure. This disclosure mayrefer to ear canals 200A, 200B collectively as “ear canals 200.”Eardrums 202A, 202B (collectively, “eardrums 202”) are located withinear canals 200. Eardrums 202 may also be referred to as tympanicmembranes. As shown in the example of FIG. 2 , EMREO sensors 106 may bepositioned at the medial tips of hearing instruments 102.

In accordance with one or more examples of this disclosure, hearinginstruments 102 may comprise housings that are customized for placementwithin ear canals 200 of user 104, up to and beyond the second bend todetect EMREOs in situ. In any of the examples of this disclosure,hearing instruments 102 may use signal pre-conditioning andpost-processing to acquire reliable EMREO estimates, as describedhereinafter with various signal processing methods.

The following section of this disclosure describes a non-limiting set ofnumbered examples of how EMREO sensors 106 may be implemented. Suchexamples may be implemented individually or in any combination.

In a first example, EMREO sensors 106A and EMREO sensors 106B eachinclude respective microphone(s) positioned at the (medial) tip ofhearing instruments 102. The microphones may measure soundwavesgenerated by EMREOs that propagate through the air in the ear canals ofuser 104. In other words, the microphones may be configured to detectchanges in air pressure within the ear canals caused by EMREOs.

It is noted that the vibrations evoked by otoacoustic emissions (OAEs)and EMREOs have different characteristics. For example, EMREOs are atlower frequencies than OAEs typically are measured, and EMREOs havelarger amplitudes than OAEs. Thus, it is possible to differentiate OAEsand EMREOs in the sounds recorded by microphones in the ear canal.Moreover, EMREOs may be reliably measured even in the presence of clicksounds that are used to elicit OAEs.

Beamforming is a technique designed to increase the relative volume ofsounds originating from a focal direction relative to other sounds.Thus, beamforming may increase the signal-to-noise ratio with respect tosounds originating from the focal direction. In general, beamformingcombines signals synchronously in time from two or more microphones suchthat correlated sounds (such as those originating from a focal direction(e.g., in front of the listener)) are enhanced and uncorrelated soundsare deemphasized (such as sounds originating from directional other thanthe focal direction). Thus, beamforming may suppress noise, where noiseis considered to be sound not originating from the focal direction. Insome examples, beamforming may be implemented in the matter described inDoclo et al., “Acoustic Beamforming for Hearing Aid Applications,”Handbook on Array Processing and Sensor Networks, Chapter 10, 2008.

In some examples, EMREO sensors 106A and EMREO sensors 106B each includean array of two or more microphones that are positioned inside the earcanals of user 104 and directed toward the eardrums of user 104.Although they may also serve other purposes, this disclosure may referto such microphones as EMREO microphones. In other words, EMREO sensors106A may include a first array of two or more EMREO microphones andEMREO sensors 106B may include a second array of two or more EMREOmicrophones. In some such examples, each of hearing instruments 102 mayapply beamforming to signals generated by the EMREO microphones of thehearing instrument with the focal direction towards the eardrums of user104. By applying beamforming, hearing instruments 102 may boost theeardrum vibration signals and reduce the impact of environment sounds.

Thus, in this example, for either or both of hearing instruments 102,the EMREO sensors 106 of the hearing instrument may include a firstmicrophone and a second microphone. The first microphone and the secondmicrophone are positioned within an ear canal of user 104 and may beconfigured to detect changes in air pressure within the ear canalscaused by EMREOs. In this example, EMREO-related measurements mayinclude a first signal produced by the first microphone and a secondsignal produced by the second microphone. One or more processors (e.g.,processors of the hearing instrument, another hearing instrument, and/orcomputing system 108) may apply a beamformer to the first signal and thesecond signal to generate a third signal. A focal direction of thebeamformer is toward the eardrum of the user of the hearing instrument.Furthermore, in this example, the processors may determine an action toperform based on the third signal.

Furthermore, hearing instruments 102 may temporally align the soundsignals from the EMREO microphones and sound signals provided toreceivers of hearing instruments 102 and then subtract the sound signalsfrom receivers of hearing instruments from the signals generated by thein-ear EMREO microphones to further reduce the impact of theenvironmental sounds. In this way, the processors may use multiplemicrophones in the ear to create a beamformer to enhance the EMREOsignal and improve the signal-to-noise ratio (SNR), where the noise isexternal sound from the environment, or biological noise such ascardiovascular or pulmonary sounds (which also are very low frequency).

Thus, in this example, for either or both of hearing instruments 102,the hearing instrument may comprise an array of external microphonesconfigured to receive sound from an environment of user 104. As part ofperforming an action based on EMREO-related measurements, a set of oneor more processors may determine a focal direction based on theEMREO-related measurements. For instance, the processors may determinethe focal direction by first determining a phase of the EMREOs (andhence a direction of eye movements) and a magnitude of the EMREOs (andhence how far the eyes moved in the direction). The processors may thendetermine the focal direction based on the direction and distance of eyemovement. Additionally, the one or more processors may apply abeamformer to signals generated by the external microphones, wherein thebeamformer has the determined focal direction. In some examples, theprocessors may steer a null of the focal direction, which is thedirection opposite the focal direction.

In a second example, EMREO sensors 106A and EMREO sensors 106B eachinclude respective medial microphone(s) positioned at the medial tips ofhearing instruments 102, hearing instruments 102 may also includelateral microphones 204A, 204B (FIG. 2 ) positioned at lateral surfaces206A, 206B (FIG. 2 ) of hearing instruments 102, such as at faceplatesof hearing instruments 102. Processors of hearing instruments 102 maycompare the signal(s) from the medial microphones and the lateralmicrophones to distinguish between ambient noise, user speech, andEMREOs. For example, the sound pressure levels (SPLs) of environmentalsounds generated outside the head of user 104 may be larger at lateralmicrophones 204 than the medial microphones. In contrast, the SPLs fromEMREOs may be larger at the medial microphones and smaller, if presentat all, at lateral microphones 204. Accordingly, the processors may usethese differences in SPLs to distinguish between environmental soundsand pressure changes caused by EMREOs. Furthermore, there may be phasedifferences between environmental sounds originating outside the head ofuser 104 and sounds caused by EMREOs. Hence, the arrival of a pressurewave at the medial microphones before lateral microphones 204 mayindicate the pressure wave is caused by EMREOs. Additionally, thecharacteristics of ambient noise, user speech and EMREOs are alldifferent when comparing the signals on the medial and lateralmicrophones from both ears. EMREOs signals may appear on the medialmicrophones and occur bilaterally (on both ears relativelysimultaneously). Ambient noise signals are typically much stronger onlateral microphones 204 than on the medial microphones and will have awide array of interaural level and timing differences. User speechtypically generates stronger signals on the medial microphones than theEMREOs and may show up synchronous signals across the ears.

Thus, in this second example, for at least one of hearing instruments102, one or more of the EMREO sensors of the hearing instrument includesa first microphone positioned at a medial tip of the hearing instrument.EMREO-related measurements generated by the EMREO sensors of the hearinginstrument may include a signal generated by the first microphone. Thehearing instrument may also comprise a second microphone positioned at alateral surface of the hearing instrument, such as a faceplate of thehearing instrument. Processors of hearing instruments 102 and/orcomputing system 108 may generate, based on a comparison of the signalgenerated by the first microphone and a signal generated by the secondmicrophone, an enhanced version of the signal generated by the firstmicrophone. For instance, the processors may compare the signalgenerated by the first microphone and the signal generated by the secondmicrophone to reduce the impact of noise (e.g., ambient noise from theenvironment of user 104, user speech sounds, or other types of noisethat may mask sounds of EMREOs) in the signal generated by the firstmicrophone. Thus, the enhanced version of the signal generated by thefirst microphone may have a greater signal-to-noise (SnR) ratio than theoriginal signal generated by the first microphone, where the signalcorresponds to air pressure changes caused by EMREOs. The processors mayperform one or more actions based on the enhanced signal. For instance,the processors may perform any of the actions set forth elsewhere inthis disclosure using the enhanced signal as a set of EMREO-relatedmeasurements.

In a third example, EMREO sensors 106A and EMREO sensors 106B eachinclude a respective vertical-cavity surface-emitting laser (VCSEL) andphotodetector positioned at the medial tips of hearing instruments 102.In other examples, other types of lasers may be used. For each ofhearing instruments 102, the VCSEL of the hearing instrument shines acoherent light beam onto one of eardrums 202 and the photodetector ofthe hearing instrument detects light reflected by the eardrum. Hearinginstruments 102 may use optical feedback interferometry based on thelight detected by the photodetector to detect EMREO motion via dopplertechniques.

Thus, in this third example, for either or both of hearing instruments102, the EMREO sensors of the hearing instrument may include a VCSELpositioned to shine a coherent beam onto the eardrum. As part ofobtaining EMREO-related measurements, processors of the hearinginstrument and/or one or more other devices (e.g., another hearinginstrument, devices of computing system 108, etc.) may apply opticalfeedback interferometry based on reflected light of the coherent beam todetermine a position of an eardrum of user 104.

In a fourth example, EMREO sensors 106A and EMREO sensors 106B eachinclude a respective Time of Flight (ToF) sensor and optical lenspositioned at the medial tips of hearing instruments 102 such that theToF sensor system has clear line of sight to the tympanic membrane. TheToF sensor may emit pulses of infrared (IR) light directed at eardrums106 and the optical lens may detect reflections of the pulses of IRlight that are reflected by eardrums 106. The amount of time requiredfor the infrared light to travel from a ToF sensor of a hearinginstrument to an eardrum and back to the optical lens of the hearinginstrument changes when the distance between the hearing instrument andthe eardrum changes. Because the eardrum moves outward and inward duringEMREOs, the hearing instrument may detect EMREOs based on changes in thedistance between the hearing instrument and the eardrum. In this way,hearing instruments 102 may detect EMREOs based on the arrival ofreflected IR energy from the tympanic membrane from an IR pulse emittedfrom the same sensor system.

Thus, in this fourth example, for either or both of hearing instruments102, the EMREO sensors of the hearing instrument may include a ToFsensor configured to emit infrared light toward the eardrum andconfigured to determine a distance to the eardrum based on a travel timeof the infrared light to the eardrum and back to the ToF sensor. In thisexample, as part of obtaining EMREO-related measurements, processors ofthe hearing instrument and/or one or more other devices (e.g., anotherhearing instrument, devices of computing system 108, etc.) may determinea position of the eardrum based on the travel time.

In a fifth example, EMREO sensors 106A and EMREO sensors 106B eachinclude a respective fixed structured light (FSL) sensor positioned atthe medial tips of hearing instruments 102. In such examples, an FSLsensor emits a fixed structured light pattern that illuminates aneardrum. The structured light pattern may comprise visible light or IRlight. The fixed structured light pattern is fixed in the sense that thestructure of the light pattern is always the same. For instance, the FSLsensor may project the same pattern of stripes onto the eardrum. The FSLsensor also includes one or more cameras that detect the pattern oflight reflected from the eardrum. The hearing instrument may use thepatterns of light detected by the one or more cameras to determine a3-dimensional shape and position of the eardrum. Because the3-dimensional shape and position of the eardrum changes during an EMREO,the hearing instrument may detect the EMREO based on changes to thedetermined 3-dimensional shape and position of the eardrum.

In a sixth example, EMREO sensors 106A and EMREO sensors 106B eachinclude a respective programmable structured light (PSL) sensorpositioned at the medial tips of hearing instruments 102. A hearinginstrument may use a PSL sensor to detect EMREOs in much the same way asdescribed elsewhere in this disclosure with respect to FSL sensors.However, the PSL sensor may be programmed to project different patternsof the structured light. In certain conditions, the PSL sensor isgenerally capable of detecting depths (i.e., displacements) on the orderof microns, whereas in some instance, the FSL sensor can only detectdisplacements on the order of a millimeter. Thus, PSL sensing may bebetter depth precision than FSL sensing.

Thus, in the fifth and sixth examples, for either or both of hearinginstruments 102, the EMREO sensors of the hearing instrument may includea structured light sensor configured to emit structured light toward theeardrum. As part of obtaining EMREO-related measurements, processors ofthe hearing instrument and/or one or more other devices (e.g., anotherhearing instrument, devices of computing system 108, etc.) may determinethe position of the eardrum based on a pattern of light detected by thestructured light sensor.

In a seventh example, EMREO sensors 106A and EMREO sensors 106B eachinclude one or more vibration sensors that are positioned in the housingof hearing instruments 102 such that the vibration sensor is in contactwith the skin of the user's ear canal. EMREO acoustic emissions on theorder of 55dBA also create surface waves that propagate from the eardrum(tympanic membrane), through the skin, and to the vibration sensors ofthe hearing instrument. Vibration sensor(s) may be of sufficientsensitivity to detect such vibrations while simultaneously rejectingacoustic noise. In other words, the vibration sensor(s) do not primarilydetect vibration caused by sounds travelling through the air, butinstead primarily detect vibration travelling through the user's skin.In some examples, the vibration sensors belong to a class of sensorsthat is based on microelectromechanical systems (MEMS) process andaluminum nitride piezoelectric layers. This class of sensors may have alower input-referred noise floor, thereby allowing sensing of lowersignals. In some examples, the vibration sensors may be implemented asdescribed in Grosh, K., et al., “Miniature implantable low noisepiezoelectric diaphragm sound sensor,” 2pEA4, J. Acoust. Soc. Amer.,Vol. 143, No. 3, Pt. 2 of 2, March 2018.

In some examples, hearing instruments 102 may correlate the signals fromthe vibration sensors with signals from other microphones of hearinginstruments 102 and/or any of the aforementioned EMREO detection sensorsto acquire reliable estimates. For instance, if a vibration sensor has astrong signal at 35 Hz while a lateral microphone (i.e., a microphone ata surface of a hearing instrument lateral to a midline of user 104) hasno 35 Hz signal, then there's a higher probability that saccadic eyemovement has occurred. In some examples, hearing instruments 102 may usemeasures of covariance across different sensors to provide a more robustestimate of the EMREO signal. For instance, in one example, correlatingthe changes in vibration from the vibrometers and the changes insound-pressure levels from the in-ear microphone, may provide a morereliable estimate of the movement of the eardrum(s) than an estimatebased on either sensor alone. The variance in the estimate is reducedowing to a larger sample size of independent measures.

Thus, in the seventh example, for either or both of hearing instruments102, the EMREO sensors of the hearing instrument may include a vibrationsensor in contact with skin of the ear canal. As part of obtainingEMREO-related measurements, processors of the hearing instrument and/orone or more other devices (e.g., another hearing instrument, devices ofcomputing system 108, etc.) may obtain measurements of surface waves inthe skin of the ear canal caused by the EMREOs of the eardrum.

In an eighth example, an implantable vibration sensor is physicallyattached to the ossicular chain or tympanic membrane of the user suchthat the processors may use output of the implantable vibration sensorto detect EMREO motion and provide detection results to a cochlearimplant or hearing instrument. In this example, the implantablevibration sensor may communicate with the hearing instrument via awireless or wired-based communication link.

Thus, in the eighth example, for either or both of hearing instruments102, the EMREO sensors of the hearing instrument may include a vibrationsensor (e.g., vibration sensor 208A, 208B of FIG. 2 ) attached to theeardrum or an ossicular chain of user 104. As part of obtainingEMREO-related measurements, processors of the hearing instrument and/orone or more other devices (e.g., another hearing instrument, devices ofcomputing system 108, etc.) may obtain, by the one or more processors,measurements of vibrations from the vibration sensor.

With the permission of user 104, processors of hearing instruments 102and/or computing system 108 may use information about the user's eyemovements for any of one or more purposes. In other words, theprocessors may perform one or more actions based on EMREO-relatedmeasurements. For instance, the processors may use EMREO-relatedmeasurements of user 104 to initiate or augment the processing behaviorsof hearing instruments 102. For instance, the processors may use theEMREO-related measurements of user 104 to supplement information fromother sensors, such as sensors in hearing instruments 102, that iscollected concurrently. The processors may use the resulting informationto make inferences about the environment of user 104, the intentions ofuser 104, physical or mental states of user 104, and so on.

The following section of this disclosure includes a nonlimiting set ofnumbered example use cases. These example use cases may be appliedindividually or in any combination.

In a first example use case, the processors may use EMREO-relatedmeasurements of user 104 to infer a user's activity, such whether theyare being social/active, or are focused/stationary. For instance, in oneexample, the processors may determine the direction of a person's voiceusing various sound localization techniques. In this example, with thepermission of user 104, the processors may determine that user 104 islikely paying attention to the person if the eyes of user 104 track thedirection of the person's voice. In some examples, the processors maydetermine that user 104 is sleeping based on the EMREO-relatedmeasurements of user 102 being consistent with the patterns of eyemovement associated with the Rapid Eye Movement (REM) phase of the sleepcycle. In some examples, the processors may determine that the user isreading based on the EMREO-related measurements of user 102corresponding to a reading pattern. The processors may provide suchinformation related to the activity of user 104 to a computing systemthat generates health-related data based on the information forpresentation to user 104 and/or one or more third-party users.

Moreover, in some examples, the processors may use EMREO-relatedmeasurements of user 102 to monitor and track sleep patterns and qualityof sleep of user 104. In other words, the processors may generate, basedon EMREO-related measurements, data regarding sleep of user 104. Forexample, rapid eye movement (REM) sleep is an important part of sleep.When a person is in REM sleep, the person's eyes move rapidly. In thisexample, the processors may use EMREO-related measurements of user 102to infer whether the user is in REM sleep.

The processors may generate various types of health-related data basedon information about the user's activity. For example, the processorsmay generate health-related data that include reports about the qualityof sleep that user 104 is getting. In some examples the processors maygenerate health-related data that provide information about how engageduser 104 is when other people are conversing.

In a second example use case, the processors may use the signals fromEMREO sensors 106 to determine whether user 104 is vocalizing. Forinstance, in addition to using signals generated by EMREO sensors 106 todetermine eye movements, the processors may also use signals generatedby EMREO sensors 106 to determine whether user 102 is vocalizing. Forinstance, in an example where EMREO sensors 106 include microphoneslocated at the medial tips of hearing instruments 102, the processorsmay compare the signals from these microphones for each ear to determinewhether sounds originated from a direction between the ears of user 104.In some examples where EMREO sensors 106 include vibration sensors, theprocessors may determine whether the user is vocalizing based onvibration signals generated by the vibration sensors. For instance,vibrations caused by vocalizations of user 104 may arrive at thevibration sensors concurrently, while vibrations caused by sound mayarrive at the left and right ears at different times. Thus, in thisexample, the processors may compare timing data of vibration signalsdetected by the vibration sensors to determine whether user 104 isvocalizing.

In a third example use case, the processors may use EMREO-relatedmeasurements of user 102 to infer information about a mental state ofuser 104, such as their level of attention. For instance, in oneexample, the processors may use EMREO-related measurements of user 104to determine whether user 104 is looking in the direction of a personwho is speaking. In another example, the processors may useEMREO-related measurements of user 104 to determine, with the permissionof user 104, whether user 104 is looking forward while walking, running,driving, or otherwise moving. This, potentially in combination withinformation about the head pose of user 104, may help determine whetheruser 104 is paying attention to what may be occurring in front of user104. The processors may provide information about the mental state ofuser 104 to a computing system that generates health-related data basedon the information for presentation to user 104 and/or one or morethird-party users.

In another example, the processors may use EMREO-related measurements ofuser 104 to determine whether user 104 is tired or fatigued. Forinstance, studies have suggested that subjective fatigue is associatedwith decreased saccadic velocity. See e.g., Di Stasi et al., “Saccadiceye movement metrics reflect surgical residents' fatigue,” Ann. Surg.Apr. 2014; 259(4); 824-9. In other words, when a person feels fatigued,the person's eyes may move slower. Because the speed of eye movements ofuser 104 may be determined from EMREOs, the processors may useEMREO-related measurements to estimate the level of fatigue of user 104.For example, the processors may establish a normative range of values ofeye-movement velocities for user 104. Subsequently, in this example, theprocessors may compare current velocities to the established normativerange of values to determine if they are indicative of a state offatigue. In some activities, the user's increased levels of fatigue arecorrelated with increased error rates. For instance, operators of heavymachinery, such as semi-trucks and mining equipment, are more likely tohave accidents if they are tired. The processors may use estimates ofthe level of fatigue of user 104 to generate warnings to user 104 and/orother persons, generate logs for regulatory purposes, or otherfunctions. It is also noted that many of the activities in which thereis a correlation between fatigue and error also involve exposure toexcessive noise. Accordingly, hearing instruments 102 may also beconfigured to provide hearing protection and/or log noise exposurelevels.

In a fourth example use case, the processors may use EMREO-relatedmeasurements of user 102 to infer salient objects in the environment ofuser 104. In other words, the processors may determine, based onEMREO-related measurements, a salient object in an environment of user104. For instance, the processors may determine, based on EMREO-relatedmeasurements, a direction of gaze of user 104. For instance, if theEMREO-related measurements indicate EMREOs corresponding to an eyemovement 45-degrees to the left of the center of user 104, theprocessors may determine that the salient object is located in adirection 45-degrees to the left of the center of user 104. In someexamples, the processors may apply a neural network or other techniquefor image recognition to identify objects. Many image recognitiontechniques are known in the art. Furthermore, in some examples, theprocessors may enable user 104 to indicate which of the one or moreidentified objects are salient. In some examples, the processors mayautomatically determine, based on one or more heuristics, which of theidentified objects are salient. For instance, the processors may use aneural network-based algorithm or business-rule based algorithm todetermine which identified objects are most likely to be consideredsalient.

For example, user 102 may wear one or more cameras that capture thefield of view of user 104. For instance, the cameras may be embedded inan MR or AR visualization device worn by user 104, embedded in eyewearworn by user 104, worn on a chest or shoulder of user 104, held in ahand of user 104, etc. In this example, the processors may determinewhich objects or areas in the images captured by the cameras are salientto user 104 based on the direction of the gaze of user 104. In someexamples, the processors may communicate information about the salientobjects or areas to one or more other devices. For instance, a devicemay present video from perspective of user 104 and visually indicate inthe video which objects or areas are salient to user 104.

In a fifth example use case, the processors may use EMREO-relatedmeasurements of user 102 to control user interfaces (e.g., volume andmemory controls, playback controls for media (e.g., play, pause, skiptrack, change channel, etc.)). In other words, the processors maydetermine user input to a user interface based on the EMREO-relatedmeasurements. For example, user 104 may wear a headset that presentsvirtual reality (VR), mixed reality (MR), or augmented reality (AR)visualizations to user 104. In this example, visualizations may includeinteractive virtual features, such as virtual menus, virtual icons,virtual buttons, and virtual objects. In examples where the headsetpresents MR or AR visualizations to user 104, user 104 may also be ableto see real objects. In this example, the processors may determine,based on the movements of the eyes of user 104 where the user is lookingat one of the virtual features or a real object. This may enable theprocessors to perform some action based on the virtual or real object atwhich user 104 is looking. For instance, in one example, the virtualfeatures may include an “up” element positioned at the right side offield of view of user 104 and a “down” element positioned at the leftside of the field of view of user 104. Thus, in this example, theprocessors may increase a volume level of hearing instruments 104 whenuser 104 looks right and decrease the volume level of hearinginstruments 104 when user 104 looks left. In another example, thevirtual features include “previous” and “next” elements and theprocessors may change with item is highlighted in a virtual menu ofitems based on whether user 104 looks left or right.

In other examples where the processors use EMREO-related measurements ofuser 102 to control user interfaces, the user interface is a voiceinterface. For instance, in one such example, one or more of hearinginstruments 102 may provide an auditory stimulus to user 104 thatprompts user 104 to look in a particular direction (e.g., left or right)in order to provide input to the processors. For instance, hearinginstruments 102 may generate sound asking user 104 to look left tochange an acoustic profile of hearing instruments 102 or to keep thecurrent acoustic profile of hearing instruments 102.

In a sixth example use case, the processors may use EMREO-relatedmeasurements of user 102 to augment the detection of falls. Falls are aleading cause of injury and death, especially among the elderly.Receiving prompt medical attention after a fall may be critical inachieving positive health outcomes for a person who has fallen.Accordingly, various techniques for automatically detecting falls havebeen developed, including techniques that use signals from one or moresensors included in hearing instruments.

For instance, in one example of the sixth use case, the processors mayimplement a machine learning system, such as an artificial neuralnetwork. In this example, the machine learning system may receiveEMREO-related measurements as input and may generate output data thatprovides information about whether user 104 has fallen. For example, theoutput data may include a confidence level that indicates a level ofconfidence that user 102 has fallen. In some examples, the output datamay indicate in a binary manner whether user 102 has fallen. In someexamples, the output data may include data indicating an estimatedseverity of the fall. The machine learning data may be trained onmatched pairs of (i) input data (which, in accordance with techniques ofthis disclosure, includes EMREO-related measurements), and (ii) dataindicating whether the input data coincided with a fall. In someexamples, the input to the machine learning system may includeEMREO-related measurements and also include other data, such as datafrom an IMU, data from a photoplethysmography sensor, data from an EKGsensor, data from one or more microphones, and so on. The machinelearning system may use all of these types of data in determining theoutput data. Including the EMREO-related measurements in the input datamay make fall detection more robust, which may result in fewer falsepositives and fewer false negatives.

In a seventh example use case, the processors may use EMREO-relatedmeasurements of user 102 to assess the susceptibility of user 102 tofalling. Research has demonstrated that there are relationships betweeneye movement patterns and errors in foot placement during walking orrunning. For example, persons who recently fell are more likely to fallagain due in part to changes in eye movements after a fall, namely lessfixation on current foot position and more fixation on upcoming objects,which may lead to errors in stepping. For instance, in this example, ifa person trips and falls on any icy sidewalk, the person may be morelikely to be looking ahead from additional icy spots and not notice achange in the angle of the sidewalk, resulting in another fall.Accordingly, in an example of the seventh use case, the processors mayuse EMREO-related measurements of user 102 to assess whether the eyemovements of user 102 are consistent with the user 102 looking forupcoming object and not looking at current foot placement.

In another example, the coordination between eye movements and footplacements was impaired in older adults who had a history of fallscompared to older adults who had not fallen previously. Accordingly, inan example of the seventh use case, the processors may use EMREO-relatedmeasurements of user 102 to determine the fall risk of user 102 based ona pattern of eye movements and foot steps of user 102.

In either example or other examples of determining the susceptibility ofuser 102 to falling, the processors may implement a machine learningsystem, such as an artificial neural network, that receives input datathat includes EMREO-related measurements and data (e.g., IMU data)indicating footfalls of user 102. In this example, the machine learningsystem may generate output data that includes data indicating asusceptibility of user 102 to falling. For instance, the output data mayinclude a fall risk score for user 102. The machine learning system maybe trained in one of a variety of ways, such as by using training datathat specifies pairs input data and correct output data.

The processors may use information about the susceptibility of user 102to falling in one or more of various ways. For instance, in someexamples, the processors may determine, based on the susceptibility ofuser 102 to falling risk, whether to notify patients, doctors, lovedones, or other people about the presence of a fall risk, or a trendtowards increased risk. The processors may store data regarding thesusceptibility of user 102 to falling for purposes of determiningwhether a trend towards increased risk is occurring.

In an eighth example use case, the processors may use EMREO-relatedmeasurements of user 102 to detect epileptic seizures or diagnoseepilepsy. In other words, the processors may detect, based onEMREO-related measurements, an epileptic seizure of user 104. Forinstance, when a person suffers an epileptic seizure, the person's eyesmay move rapidly. Furthermore, people with epilepsy may have patterns ofeye movements that are different from people who do not have epilepsy.Hence, the processors may use EMREO-related measurements of user 102 todetermine whether user 104 is experiencing an epileptic seizure orpossibly has epilepsy.

In a ninth example use case, the processors or humans may useEMREO-related measurements of user 104 to supplement informationprovided by concurrent and covarying signals collected by hearinginstruments 102 or other sensor devices, such as acoustic information,patterns of vibration on the skin of the ear canal,electroencephalography (EEG), electrooculography (EOG), accelerometerdata, gyroscope data, etc.

For instance, in the ninth use case, the processors or humans may useEMREO-related measurements of user 102, EEG signals, respiration rate,respiration patterns, heart rate, heart rhythms, vocalizations of user102, IMU data (e.g., accelerometer data and gyroscope data), snoring,body temperature, environmental temperature, environmental humidity,environmental noise, environmental light levels, and other informationto detect sleep problems. For instance, the processors of hearinginstruments 102 and/or computing system 108 may use such information asinput to a neural network algorithm or business rule-based algorithmthat diagnoses sleep problems.

Sleep problems are a common complaint. Diagnosing sleep problemstypically requires a patient to sleep at a sleep clinic that hasspecialized equipment. Visits to sleep clinics are typically expensiveand disruptive to the patient's routine. Moreover, for many patients,the type of sleep the patients have at a sleep clinic is notrepresentative of the type of sleep the patients have at home because ofthe unfamiliar setting of sleep clinic, because of the feeling of beingobserved, and so on. However, much of the information collected during avisit to a sleep clinic may be collected instead by sensors in hearinginstruments 102. Thus, a patient (e.g., user 104) may take hearinginstruments 102 home and sleep there for one or more nights instead ofat a sleep clinic. This may enable more healthcare professionals to getmore representative data regarding the sleep patterns of the patient.Being able to get more representative data regarding the sleep patternsof the patient may be especially useful in identifying and treatingsleep disorders that happen only occasionally, such as somnambulism andnight terrors. Gathering information about REM sleep is one component ofperforming a sleep exam. As discussed elsewhere in this disclosure,EMREO-related measurements may be used to detect REM sleep. Thus, inthis way, the one or more processors may generate, based onEMREO-related measurements, data regarding sleep of user 104, such asthe lengths of time user 104 spends in REM sleep/non-REM sleep, times ofonset of REM sleep/non-REM sleep. The data regarding sleep of user 104may include data based on other sensors in hearing instruments 102, suchas IMU signals for determining restlessness during sleep, breathingrates and patterns (e.g., for detecting sleep apnea), microphone signalsfor detecting snoring, and so on.

In another example of the ninth use case, the processors or humans mayuse EMREO-related measurements of user 104 to conduct sobriety testing(e.g., in the context of traffic stops). In this example, impairedpersons may exhibit horizontal gaze nystagmus and poor balance.Horizontal gaze nystagmus is a type of involuntary jerking eye movement.EMREO sensors 106 may generate EMREO-related measurements that provideempirical evidence of horizontal gaze nystagmus. Additionally, IMUs inhearing instruments 102 may generate empirical evidence of balanceproblems. Generation of such empirical evidence may reduce the relianceof the subjective judgment of law enforcement officers. In someexamples, the processors may apply one or more algorithms (e.g., neuralnetwork algorithms, business rule algorithms, etc.) that take as inputEMREO-related measurements of user 104 and other input data to generatescores or other output to indicate results of a sobriety test. In someexamples, other types of data are not used in the sobriety test.

In a tenth example use case, the processors may use EMREO-relatedmeasurements of user 104 to adaptively change the signal-processingbehavior of cochlear implants by detecting vibration on the ossicularchain or on the skin of the ear canal. For example, many of use casesdescribed in this disclosure may be applied with respect to cochlearimplants, including application of beamforming with steering of a focaldirection (directional beam) in a direction of eye fixation, or theadjustment of volume based on the gaze behavior of the user.

In an eleventh example use case, the processors may use EMREO-relatedmeasurements of user 104 to determine the placement of an allocentricsound source. An allocentric sound source is a sound source thatoriginates from a location near the center of a group of people, as isfrequently the case for theatres and concert venues. Consider forexample a situation in which an orchestra is performing on a stage.Typically, in such situations, there is a set of microphones positionednear or above the orchestra to detect sound generated by performers ofthe orchestra. The microphones are generally positioned and balanced tocapture sound as that sound would be perceived by a person centered infront of the stage. Signals generated by the microphones may betransmitted to hearing instruments 102. In this way, user 104 may beable to better hear the sound generated by the orchestra than whenhearing instruments 102 process sound detected by microphones of hearinginstruments 102. However, user 104 may not actually be centered in frontof the orchestra. Moreover, user 104 might not always have their headturned to look at the orchestra. Thus, user 104 may perceive the soundsproduced by the orchestra to always originate from a point in space infront of user 104, regardless of where user 104 is with respect to theorchestra and regardless of the direction in which user 104 is looking.This may be disorienting to user 104 or may reduce the enjoyment of user104.

Hearing instruments 102 may be configured to correct this problem bymodifying a left-ear signal and a right-ear signal. For instance, in theexample above, if the orchestra is to the left of user 104, hearinginstruments 102 may introduce a delay and reduce the intensity of theright-ear signal, thereby causing user 104 to perceive the sound of theorchestra to be coming from the left of user 104. The processors maydetermine the amount of delay and/or intensity modification to introduceusing mathematical models based on the angle of origin of the soundrelative to a sagittal plane of user 104. For instance, the processorsmay apply Head-Related Transfer Functions (HRTFs) that describe therelative relationship of sound signals hitting the two eardrums for anygiven sound-source location relative to the listener.

However, there are challenges associated with techniques for correctingthe sound in this manner. For instance, requiring user 104 to manuallyindicate the angle of their head relative to the actual sound source maybe time consuming and awkward for user 104. Furthermore, hearinginstruments 102 may be able to determine the angle of the head of user104 relative to the actual sound source based on IMU signals related tothe movement of the head of user 104. However, because of the timerequired to determine this angle based on the IMU signals, there may bea delay between when user 104 moves their head and when hearinginstruments 102 are able to modify the left-ear and right-ear signals tocompensate for the change in angle between the head of user 104 and theactual sound source. This may result in user 104 feeling that theactual, visually-perceived origin point of the sound of the orchestra isnot synchronized with the audibly-perceived origin point of the sound ofthe orchestra as perceive. In fact, user 104 may get the sensation thatthe audibly-perceived origin point of the sound is chasing after theactual, visually-perceived origin point of the sound.

It is noted that people generally move their eyes in a direction beforeturning their heads to look in that directions. Additionally, it isnoted that during an EMREO, the eardrums begin to vibrate before theeyes actually move. Thus, there is a time delay between when theeardrums being to vibrate and when the head of user 104 actually moves.This time delay may provide sufficient time for the processors todetermine how to modify the left-ear signal and the right-ear signalbefore the user actually turns their head to toward the sound source.Determining how to modify the left-ear signal and the right-ear signalbefore the user actually turns their head to toward the sound source mayimprove the experience of the user because it may help to eliminate thedelay between the time when the user turns their head and the time whenhearing instruments 102 start modifying the left-ear signal and theright-ear signal. In this way, user 104 may not get the sensation thatthe audibly-perceived origin point of the sound is chasing after theactual, visually-perceived origin point of the sound as user 104 turnstheir head. It is noted that user 104 might glance to one side but notactually turn their head to follow the glance. Nevertheless, it may beworthwhile determining how to modify the left-ear and right-ear signalsin case user 104 does follow through and move their head.

Thus, in some such examples, the processors may determine, based on theEMREO-related measurements, potential modifications to left-ear andright-ear audio signals. Additionally, the processors may apply thepotential modifications to the left-ear and right-ear audio signals inresponse to determining a movement of a head of the user. In suchexamples, it may be assumed that the eyes of user 104 are oriented inthe direction of attention of user 104, and that is where attendedsounds originate. By inferring this, the polar pattern of directionalhearing aids can be steered so that sounds coming from that directionare emphasized and sounds from other directions are deemphasized (e.g.,by applying beamforming).

In a twelfth example use case, the processors may use EMREO-relatedmeasurements of user 104 to select a directional processing mode. Forexample, users of hearing instruments commonly struggle to understandspeech in loud ambient noise and may rely on reading a person's lips toaugment intelligibility. This is often referred to as the ‘cocktailparty’ effect. To address this cocktail party effect, hearinginstruments 102 may be configured to use directional processing modes toattenuate sounds arriving from locations other than a location of aperson to whom user 104 wants to listen. For instance, if the person towhom user 104 wants to listen is directly in front of user 104, hearinginstruments 102 may attenuate sounds arriving from all other directionsthan directly in front of user 104. Similarly, if the person to whomuser 104 wants to listen is directly in left of user 104, hearinginstruments 102 may attenuate sounds arriving from all other directionsthan to the left of user 104.

Hearing instruments 102 have traditionally relied upon receivingindications of user input to indicate whether user 104 is in a situationin which a directional processing mode is desired and to indicate thedirection of the sounds to which user 104 wants to listen.Alternatively, hearing instruments 102 may rely on the head movements ofuser 104 to determine a direction of the sounds to which user 104 wantsto listen. Similar to the examples provided above with respect toallocentric sounds, providing such input or relying on head movementsmay introduce distracting delays. Moreover, relying on head movementsfor directional processing may require user 104 to actually move theirhead to the direction of the person to whom user 104 wants to listen. Inreal life, people do not always turn their head to look at the person towhom they want to listen. Rather, people frequently just move their eyesto look at the person to whom they want to listen.

As noted above, during EMREOs, the eardrums may begin to move prior tothe eyes of user 104 beginning to move. Hence, the processors may havesufficient time to determine how to adapt the directional processingmode and then to adapt the directional processing mode accordingly toenhance the perceptibility of sounds arriving from the direction thatuser 104 is about to look. Thus, if multiple people are speaking arounduser 104, detecting eye movements of user 104 via EMREOs may provideinsight into optimal settings for hearing instruments 102. For instance,in this example, if the EMREOs of user 104 suggest that user 104 islooking to the right or about to look to the right, hearing instruments102 may use a directional processing mode to attenuate sounds arrivingfrom directions other than the rightward direction. In any of theexamples of this disclosure, the processors may use EMREO data to steera microphone beamformer, or to select from a set of pre-designedbeamformers, or to determine a primary lobe width of an adaptivebeamformer. For instance, in one example, the processors may determinethe primary lobe width based on the range of eye movements over a setamount of time over which that calculation is made, the range being thedistance along the azimuth between the leftmost eye movement and therightmost eye movement. In the examples above that use pre-designedbeamformers, each of the pre-designed beamformers may have differentfocal directions and/or primary lobe widths. In this way, beamformersteering may begin before the eyes move, thus reducing the latencybetween eye movement and beam steering.

In a thirteenth example use case, the processors may use EMREO-relatedmeasurements of user 104 to facilitate interpersonal communication.Thus, the one or more processors may determine an interpersonal messagebased on EMREO-related measurements and may send the interpersonalmessage. Using EMREO-related measurements of user 104 to facilitateinterpersonal communication may be especially valuable in situationswhere it is impractical or undesirable to communicate verbally, manually(e.g., with hand gestures), keyboards, or touchscreens. Accordingly,hearing instruments 102 may be configured for wireless communication (ormay be communicatively coupled using a wireless- or non-wirelesscommunication technology to another device that is configured forinter-device communication). In this example, user 104 may perform aseries of eye movements that correspond to a message that user 104 wantsto convey. EMREO sensors 106 may detect EMREOs corresponding to theseries of eye movements. The processors may then cause hearinginstruments 102 to send data representing the message.

Sending a message in this way may be useful in any of a variety ofscenarios. Consider, for example, a firefighter who is battling a fire.The sound of the fire and their equipment may prevent verbalcommunication and the firefighter might have their hands full withfirefighting equipment. In this example, the firefighter wearing hearinginstruments 102 may use a series of intentional eye movements togenerate a message that hearing instruments 102 communicate to devicesassociated with one or more other users, such as the firefighter'scolleagues.

User 104 may use eye movements to generate a message in one or more ofvarious ways. For instance, in one example, user 104 may use a series ofglances that encode information in a manner similar to Morse code. Morsecode is a character encoding scheme that uses sequences of differentsignal durations (dots and dashes) to represent characters, such asletters and numbers. In this example, short glances in a direction maycorrespond to dots and long glances in the same direction may correspondto dashes. In another example, glances to the right may correspond todots and glances to the left may correspond to dashes. In some examples,sequences of long or short glances, glances left or right, orcombinations thereof may be used to encode characters, full words,phrases, or concepts.

In another example of how user 104 may use eye movements to generate amessage, a user interface presented to user 104 (e.g., a voiceinterface, an MR visualization, etc.) may provide user 104 with a set ofoptions. These options may include words, phrases, categories, etc. User104 may perform eye movements to select one or more options in order toform a message. In some examples, the processors may use predictiontechniques to select the options. For instance, given that user 104selected an option corresponding to a first word, the processors mayprovide options indicating words that are likely to follow the firstword. Such prediction techniques are commonly found in smartphonevirtual keyboards.

One or more of the use cases described in this disclosure may findapplication in various scenarios. For instance, in one example, considerthat user 104 may be a member of a group. The people in the group maynot want to speak for any of a variety of reasons. At the same time, thepeople may be unable to see hand gestures because of obstacles, or maybe unable to use hand gestures, keyboards, or touchscreens because theirhands are otherwise occupied. In this scenario, user 104, may perform anintentional series of eye movements that correspond to a message thatuser 104 wants to convey to the other people in the group. For instance,user 104 may want to convey a message about their plans or request help.In this example, hearing instruments 102 may export data correspondingto the message for wireless transmission. In some examples, devicesreceiving the message may use visual, audible, and/or tactile/hapticoutput to present the message to persons associated with the receivingdevices.

Temporary or permanent hearing loss caused by loud sounds is an issueconfronting people in a variety of occupations. For instance, the soundsof the equipment used by or near user 104 may prevent the person fromreceiving audibly-conveyed information. At the same time, conventionalearplugs or sound protection ear muffs may also prevent user 104 fromhearing audibly-conveyed commands or from hearing the movements ofallies and enemies. Moreover, exposure to loud sounds may lead totinnitus. Tinnitus and permanent hearing loss impose significantfinancial burdens on healthcare systems. Accordingly, earplugs havepreviously been introduced that provide hearing protection by reducinghigh-decibel sounds while allowing the users to hear lower-decibelsounds. Advantageously, hearing instruments 102 that offer such hearingprotection may also detect EMREOs and use information about users' eyemovements (e.g., EMREO-related measurements) for any of the purposesdescribed in this disclosure. For instance, hearing instruments 102 mayoffer hearing protection and enable users to send interpersonalmessages.

Furthermore, consider a scenario as discussed above with respect to thefourth example use case where the processors may use EMREO-relatedmeasurements of user 102 to determine salient objects or areas in theenvironment of user 104. In this scenario, a group of users may wear orotherwise use MR visualization devices and cameras that capture thefields of views of the users. It may be difficult for a first user toquickly explain the location of an object of interest to her fellowusers. In accordance with a technique of this disclosure, the MRvisualization devices of fellow users may include windows showing thefirst user's field of view. The first user may fixate her gaze on theobject of interest. EMREO sensors 106 may detect signals of EMREOsassociated with the first user moving her eyes to fixate on the objectof interest. The processors may determine, based on EMREO-relatedmeasurements of the first user, a location of the object of interestwithin the first user's field of view and mark that location within thewindows presented by the MR visualization devices of the other users. Inthis example, the first user may also use eye gestures as previouslydiscussed to communicate information to her follow users.

Communication security is also of great importance in some scenarios.For example, a user may obtain a communication tool, such as a radio,belonging to another user. In this scenario, the user may use thecommunication tool inappropriately. Techniques of this disclosure mayreduce the risk of this scenario occurring. For instance, as describedelsewhere in this disclosure, hearing instruments 102 may include avariety of sensors, including EMREO sensors 106 and may be used to sendinterpersonal messages. Should the processors determine that the signalsfrom such sensors represent the presence or absence of particular typesof signals (e.g., absence of heart rate signal, absence of EMREOs,etc.), hearing instruments 102 may disable one or more capabilities(e.g., inter-personal communication capabilities) of hearing instruments102, thereby preventing hearing instruments 102 from being used by aperson other than an authorized user.

In accordance with some examples of this disclosure, in-ear and/oraround-the-ear electrodes may be used employed together withEMREO-related measurements to more robustly detect and track eyemovements. For example, electrooculography (EOG) is a technique in whichelectrodes placed on a person's head or face may detect electricalsignals associated with eye movements. In general, the electricalsignals contain more noise when the electrodes are placed within theperson's ears and contain less noise when the electrodes are placedcloser to the person's eyes. Thus, it may be more difficult to determinethe person's eye movements based on signals from electrodes that areplaced within the person's ears than from electrodes that are placedclose to the person's eyes. However, placement of electrodes close to aperson's eyes has negative practical and aesthetic consequences. Forinstance, electrodes positioned on a person's face are easily knockedoff during activities or during sleep and may not have an acceptableappearance for typical use.

FIG. 3 is a conceptual diagram illustrating example hearing instruments102 having EMREO sensors 106 and EOG electrodes 300A, 300B, inaccordance with one or more techniques of this disclosure. Thisdisclosure may refer to EOG electrodes 300 collectively as “EOGelectrodes 300.” In the example of FIG. 3 , the processors of hearinginstruments 102 and/or computing system 108 may use signals from the EOGelectrodes 300 and also EMREO-related measurements from EMREO sensors106 to generate information about the eye movements of user 104. Forinstance, in this example, EMREO sensors 106 may include microphones,VCSELs, ToF sensors, FSL sensors, PSL sensors, vibration sensors,implantable vibration sensors, or other types of sensors designed togenerate EMREO-related measurements.

As shown in the example of FIG. 3 , EOG electrodes 300 are positioned inthe ears of user 104. As noted above, the electrical signals generatedby in-ear EOG electrodes may be noisier than electrical signalsgenerated by EOG electrodes closer to a person's eyes. However, inaccordance with a technique of this disclosure, this problem may beovercome by using EMREO-related measurements to confirm eye movementsdetected using signals from in-ear EOG electrodes 300, or vice versa.For instance, if both the signals from EOG electrodes 300 and EMREOsensors 106 indicate an eye movement, there may be higher confidencethat the eye movement in fact occurred. This may make the system as awhole more robust.

Furthermore, in some examples of this disclosure, the processors may useEMREO data in conjunction with data from EOG electrodes 300 and/or othersensors to provide a reference angle for one or more gyroscopes of oneor more of hearing instruments 102.

Vibrometry is a technique for non-contact measurement and imaging ofvibration of a surface. In the context of detecting eye movements, thesurface may be the eardrums of user 104 or the skin of the external earcanal. The processors of hearing instruments 102 and/or computing system108 may use vibrometry to detect oscillatory pressure waves caused byEMREOs on the skin of the external ear canal of user 104 in isolationof, or in conjunction with, sound-pressure level changes of EMREOs toselect actions to perform, such as initiating or augmenting changes inthe behavior of one or more of hearing instruments 102. For instance,the processors may use vibrometry to detect oscillatory pressure wavescaused by EMREOs on the skin of the external ear canal or on theossicular chain in isolation of, or in conjunction with, thesound-pressure level changes of EMREOs to initiate or augment changes inthe behavior of a cochlear implant or other applications described inthis disclosure.

EMREOs are likely triggered by contractions of the middle-ear musclesand subsequent flexing of the ossicular chain. Contraction of the middleear muscles causes a damping of the transmission of sound to the innerear, thereby attenuating incoming sounds. This occurs reflexively duringloud sounds, presumably as a protective mechanism. Accordingly, in someexamples, the processors may use EMREOs and/or vibrometry to limit theoutput of one or more of hearing instruments 102 when sound-evokedmiddle-ear muscle contractions are detected. For instance, theprocessors may reduce the gain of incoming sound and/or apply noisecancelation to further attenuate incoming sound.

In accordance with any of the examples of this disclosure, theprocessors of hearing instruments 102 and/or computing system 108 maycreate a mapping between sets of input data and one or more types ofEMREO-related measurements of user 104. For instance, the types ofEMREO-related measurements of user 104 may include the direction of theeye movements, magnitude of the eye movements, speed of the eyemovements, and so on. The input data may include EMREO data (e.g.,EMREO-related measurements or data generated based on EMREO-relatedmeasurements). In some examples, the input data may also include othertypes of data, such as EOG data. This mapping may increase thelikelihood that the processors may detect individual eye movements moreor less in real-time.

In some examples, the processors may use machine-learning to generate amapping between the input data and the one or more types of informationabout the eye movements of user 104 (e.g., EMREO-related measurements).For instance, in some examples, the processors may implement anartificial neural network that takes the input data as an input vectorand generates an output vector containing information about the eyemovements of user 104. The neural network may be trained in one ofvarious ways. For instance, in some examples, the neural network mayinitially be trained based on training data that includes input data andcorresponding information about the eye movements of a plurality ofpeople. In some such examples, to customize hearing instruments 102 touser 104, the neural network may later be refined based on input dataand corresponding information about the eye movements of user 104. Inother examples, the neural network may be trained only on input data andinformation about the eye movements of user 104.

Thus, ground-truth eye movement data may be collected via traditionalcamera-based or EOG approaches while synchronous EMREO data is recorded.The processors may use the ground-truth eye movement data as trainingdata for machine learning algorithms that later predict informationabout the eye movements of user 104. With these data, a machine-learningalgorithm may be trained to learn the relationship between EMREO dataand eye movement. Training data may be collected across multiple usersand multiple sessions for individual users to increase the robustness ofthe algorithm and the generalization of the solution to novel users,slight changes in hearing-aid position and different environmentalcontexts.

Furthermore, in some examples, user 104 may be able to help train themachine learning algorithms by providing feedback. This may beespecially helping in the context of over-the-counter ordirect-to-consumer hearing instruments. In such examples, while user 104is wearing hearing instruments 102, user 104 may be prompted (e.g., byhearing instruments 102, another device, or a person) to move their eyesin particular directions and EMREO sensors 106 may generatecorresponding EMREO-related measurements. Furthermore, in some examples,user 104 may provide feedback if the processors of hearing instruments102 and/or computing system 108 select an incorrect action based oninformation about the eye movements of user 104. For instance, if theaction selected by the processors corresponds to a rightward eyemovement, but user 104 actually moved their eyes leftward or did notmove their eyes, user 104 may provide feedback to the processors, whichthe processors may apply as another piece of training data. By providingfeedback in this manner, it may be unnecessary for user 104 toparticipate in a training session with a hearing professional in orderto train the machine learning algorithms to generate information aboutthe movements of the eyes of user 104.

In some examples, the processor may use signal-processing techniques togenerate a mapping between the input data and the one or more types ofinformation about the eye movements of user 104. For instance, in someexamples, the signal-processing techniques may include be machinelearning models that map of signals (which may include EMREO-relatedmeasurements) to a particular eye movement. Such machine learning modelsmay be trained according to standard machine learning techniques, suchas backpropagation. In some examples, the processors may implement athreshold-based method in which the processors calculate and evaluate astatistic or set of statistics of the EMREO-related measurements todetermine a particular eye movement. For instance, the processors maydetermine, based on a statistic regarding the EMREO-related data beinggreater than a particular threshold, that the eyes of user 102 have movea particular distance in a particular direction.

The processors of hearing instruments 102 and/or computing system 108may use EMREO sensors 106 in one ear or both ears of user 104 to makeEMREO-related measurements. Because both of eardrums 202 (FIG. 2 )oscillate in sync when the eyes of user 104 move and because theinformation across that ears may be redundant, the processors may detectEMREOs using EMREO-related measurements from EMREO sensors in only asingle ear of user 104. For instance, the processors may detect EMREOsbased on EMREO-related measurements made by EMREO sensors 106A (and notEMREO sensors 106B, if EMREO sensors 106B are even included in hearinginstrument 102B).

In other examples of this disclosure, EMREO sensors 106A and EMREOsensors 106B may generate EMREO-related measurements for both ears ofuser 104. For instance, in one such example, hearing instrument 102A maygenerate first EMREO data based on EMREO-related measurements generatedby EMREO sensors 106A. In this example, hearing instrument 102A maycommunicate the first EMREO data to hearing instrument 102B.Furthermore, in this example, hearing instrument 102B may use the firstEMREO data and EMREO-related measurements generated by EMREO sensors106B to determine one or more actions. In this example, hearinginstrument 102B may send instructions to hearing instrument 102A toperform the one or more actions. For instance, hearing instrument 102Bmay instruct hearing instrument 102A to adjust use a directionalprocessing mode. In this example, the roles of hearing instrument 102Aand hearing instrument 102B may be reversed.

In another example where EMREO sensors 106A and EMREO sensors 106Bgenerate EMREO-related measurements for both ears of user 104, hearinginstrument 102A may generate first EMREO data based on EMREO-relatedmeasurements generated by EMREO sensors 106A. Additionally, in thisexample, hearing instrument 102B may generate second EMREO data based onEMREO-related measurements generated by EMREO sensors 106B. In thisexample, hearing instrument 102A may communicate the first EMREO data tohearing instrument 102B and hearing instrument 102B may communicate thesecond EMREO data to hearing instrument 102A. Furthermore, in thisexample, hearing instrument 102B may use the first EMREO data andEMREO-related measurements generated by EMREO sensors 106B to determineand perform one or more actions. In this example, because each ofhearing instruments 102 is able to determine the actions based on thesame EMREO data, hearing instruments 102 may be able to identify andperform the actions without one of hearing instruments 102 instructingthe other one of hearing instruments 102 to do so. For instance, hearinginstruments 102 may each adjust to use a directional processing modewithout one of hearing instruments 102 instructing the other to do so.

In another example where EMREO sensors 106A and EMREO sensors 106Bgenerate EMREO-related measurements for both ears of user 102, hearinginstrument 102A may generate first EMREO data based on EMREO-relateddata generated by EMREO sensors 106A. Additionally, in this example,hearing instrument 102B may generate second EMREO data based onEMREO-related data generated by EMREO sensors 106B. In this example,hearing instrument 102A may communicate the first EMREO data tocomputing system 108 and hearing instrument 102B may communicate thesecond EMREO data to computing system 108. Furthermore, in this example,computing system 108 may determine one or more actions. Computing system108 may then instruct either or both of hearing instruments 102 toperform the actions.

Thus, in these examples, a set of processors may include a first set ofprocessors and a second set of processors, where the first set ofprocessors is in hearing instrument 102A and the second set ofprocessors is in hearing instrument 102B. The first set of processorsmay receive first EMREO-related measurements from one or more EMREOsensors 106A of hearing instrument 102A. The second set of processorsmay receive second EMREO-related measurements from one or more EMREOsensors 106B of hearing instrument 102B. The one or more processors mayperform an action, such as any of the actions described in examples ofthis disclosure, based on the first EMREO-related measurements and thesecond EMREO-related measurements. For instance, the processors may usethe first EMREO-related measurement to confirm EMREOs corresponding tothe second EMREO-related measurements, or vice versa. In some examples,a communication interface of the second hearing instrument may transmitdata based on the second EMREO-related measurements to the first hearinginstrument.

In any of the examples of communication between hearing instruments 102,the communication between hearing instrument 102A and hearing instrument102B may be directly between hearing instrument 102A and hearinginstrument 102B. In examples where hearing instruments 102 communicatedirectly, the communication between hearing instruments 102 may bewireless or non-wireless. Alternatively, in any of the examples ofcommunication between hearing instruments 102, the communication betweenhearing instrument 102A and hearing instrument 102B may occur indirectlyvia one or more other computing devices or systems, such as computingsystem 108 (FIG. 1 ). In some examples where hearing instruments 102communicate with computing system 108, one of hearing instruments 102may communicate with computing system 108 by way of the other one ofhearing instruments 102. In other examples where hearing instruments 102communicate with computing system 108, each of hearing instruments 102may communicate independently with computing system 108.

In examples where processors one of hearing instruments 102 receivesEMREO data from the other one of hearing instruments 102 or processorsof computing system 108 receives EMREO data from both of hearinginstruments 102, the processors may apply signal processing techniquesto enhance the EMREO data. This may increase the level of confidence inthe determined presence and characteristics of EMREO. For instance, theprocessors may apply cross-correlation techniques for more reliabledetection and to speed the analyses of the signals for faster changes toone or more of hearing instruments 102 based on the data.

FIG. 4 is a block diagram illustrating example components of hearinginstrument 400, in accordance with one or more aspects of thisdisclosure. Hearing instrument 400 may be either one of hearinginstruments 102. In the example of FIG. 4 , hearing instrument 400comprises one or more storage devices 402, one or more communicationunit(s) 404, a receiver 406, one or more processor(s) 408, one or moremicrophone(s) 410, a set of sensors 412, a power source 414, and one ormore communication channels 416. Communication channels 414 providecommunication between storage devices 402, communication unit(s) 404,receiver 406, processor(s) 408, a microphone(s) 410, and sensors 412.Components 402, 404, 406, 408, 410, and 412 may draw electrical powerfrom power source 414.

In the example of FIG. 4 , each of components 402, 404, 406, 408, 410,412, 414, and 416 are contained within a single housing 418. However, inother examples of this disclosure, components 402, 404, 406, 408, 410,412, 414, and 416 may be distributed among two or more housings. Forinstance, in an example where hearing instrument 400 is a RIC device,receiver 406 and one or more of sensors 412 may be include in an in-earhousing separate from a behind-the-ear housing that contains theremaining components of hearing instrument 400. In such examples, a RICcable may connect the two housings.

Furthermore, in the example of FIG. 4 , sensors 412 include an IMU 426that is configured to generate data regarding the motion of hearinginstrument 400. IMU 426 may include a set of sensors. For instance, inthe example of FIG. 4 , IMU 426 includes one or more of accelerometers428, a gyroscope 430, a magnetometer 432, combinations thereof, and/orother sensors for determining the motion of hearing instrument 400.

Additionally, in the example of FIG. 4 , sensors 412 also include EMREOsensors 434. EMREO sensors 434 may be either EMREO sensors 106A or EMREOsensors 106B, as described elsewhere in this disclosure. Furthermore, inthe example of FIG. 4 , hearing instrument 400 may include one or moreadditional sensors 436. Additional sensors 436 may include EOGelectrodes 300A, 300B (FIG. 3 ), a photoplethysmography (PPG) sensor,blood oximetry sensors, blood pressure sensors, electrocardiograph (EKG)sensors, body temperature sensors, electroencephalography (EEG) sensors,environmental temperature sensors, environmental pressure sensors,environmental humidity sensors, skin galvanic response sensors, and/orother types of sensors. In other examples, hearing instrument 400 andsensors 412 may include more, fewer, or different components.

Storage devices 402 may store data. Storage devices 402 may comprisevolatile memory and may therefore not retain stored contents if poweredoff. Examples of volatile memories may include random access memories(RAM), dynamic random access memories (DRAM), static random accessmemories (SRAM), and other forms of volatile memories known in the art.Storage devices 402 may further be configured for long-term storage ofinformation as non-volatile memory space and retain information afterpower on/off cycles. Examples of non-volatile memory configurations mayinclude magnetic hard discs, optical discs, floppy discs, flashmemories, or forms of electrically programmable memories (EPROM) orelectrically erasable and programmable (EEPROM) memories.

Communication unit(s) 404 may enable hearing instrument 400 to send datato and receive data from one or more other devices, such as anotherhearing instrument, an accessory device, a mobile device, or anothertypes of device. Communication unit(s) 404 may enable hearing instrument400 using wireless or non-wireless communication technologies. Forinstance, communication unit(s) 404 enable hearing instrument 400 tocommunicate using one or more of various types of wireless technology,such as a BLUETOOTH™ technology, 3G, 4G, 4G LTE, 5G, ZigBee, WI-FI™,Near-Field Magnetic Induction (NFMI), ultrasonic communication, infrared(IR) communication, or another wireless communication technology. Insome examples, communication unit(s) 404 may enable hearing instrument400 to communicate using a cable-based technology, such as a UniversalSerial Bus (USB) technology.

Receiver 406 comprises one or more speakers for generating audiblesound. Microphone(s) 410 detects incoming sound and generates one ormore electrical signals (e.g., an analog or digital electrical signal)representing the incoming sound.

Processor(s) 408 may be processing circuits configured to performvarious activities. For example, processor(s) 408 may process the signalgenerated by microphone(s) 410 to enhance, amplify, or cancel-outparticular channels within the incoming sound. Processor(s) 408 may thencause receiver 406 to generate sound based on the processed signal. Insome examples, processor(s) 408 include one or more digital signalprocessors (DSPs). In some examples, processor(s) 408 may causecommunication unit(s) 404 to transmit one or more of various types ofdata. For example, processor(s) 408 may cause communication unit(s) 404to transmit data to computing system 108. Furthermore, communicationunit(s) 404 may receive audio data from computing system 108 andprocessor(s) 408 may cause receiver 406 to output sound based on theaudio data.

In accordance with one or more techniques of this disclosure,processor(s) 408 may obtain one or more signals generated by one or moresensors (e.g., EMREO sensor(s) 434 and, in some examples, one or moreadditional sensors 436) of hearing instrument 400 that are locatedwithin an ear canal of a user (e.g., user 104) of hearing instrument400. Processor(s) 408 may perform one or more actions based on the oneor more signals being indicative of an occurrence of EMREOs of aneardrum of the user of hearing instrument 400. For instance,processor(s) 408 may perform actions as describe in any of the examplesprovided elsewhere in this disclosure.

In some examples, hearing instrument 102 is a “plug-n-play” type ofdevice. In some examples, hearing instrument 102 is programmable to helpthe user manage things like wind noise. Furthermore, in some examples,hearing instrument 102 comprises a custom earmold or a standard receivermodule at the end of a RIC cable. The additional volume in a customearmold may allow room for components such as sensors (accelerometers,heartrate monitors, temp sensors), a woofer-tweeter, and an acousticvalve that provides occlusion when desired. In some examples, a sixconductor RIC cable is used for in hearing instruments with sensors,woofer-tweeters, and/or acoustic valves.

FIG. 5 is a block diagram illustrating example components of computingdevice 500, in accordance with one or more aspects of this disclosure.FIG. 5 illustrates only one particular example of computing device 500,and many other example configurations of computing device 500 exist.Computing device 500 may be a computing device in computing system 108(FIG. 1 ).

As shown in the example of FIG. 5 , computing device 500 includes one ormore processor(s) 502, one or more communication unit(s) 504, one ormore input device(s) 508, one or more output device(s) 510, a displayscreen 512, a power source 514, one or more storage device(s) 516, andone or more communication channels 518. Computing device 500 may includeother components. For example, computing device 500 may include physicalbuttons, microphones, speakers, communication ports, and so on.Communication channel(s) 518 may interconnect each of components 502,504, 508, 510, 512, and 516 for inter-component communications(physically, communicatively, and/or operatively). In some examples,communication channel(s) 518 may include a system bus, a networkconnection, an inter-process communication data structure, or any othermethod for communicating data. Power source 514 may provide electricalenergy to components 502, 504, 508, 510, 512 and 516.

Storage device(s) 516 may store information required for use duringoperation of computing device 500. In some examples, storage device(s)516 have the primary purpose of being a short term and not a long-termcomputer-readable storage medium. Storage device(s) 516 may be volatilememory and may therefore not retain stored contents if powered off.Storage device(s) 516 may further be configured for long-term storage ofinformation as non-volatile memory space and retain information afterpower on/off cycles. In some examples, processor(s) 502 on computingdevice 500 read and may execute instructions stored by storage device(s)516.

Computing device 500 may include one or more input device(s) 508 thatcomputing device 500 uses to receive user input. Examples of user inputinclude tactile, audio, and video user input. Input device(s) 508 mayinclude presence-sensitive screens, touch-sensitive screens, mice,keyboards, voice responsive systems, microphones or other types ofdevices for detecting input from a human or machine.

Communication unit(s) 504 may enable computing device 500 to send datato and receive data from one or more other computing devices (e.g., viaa communications network, such as a local area network or the Internet).For instance, communication unit(s) 504 may be configured to receivesource data exported by hearing instrument(s) 102, receive comment datagenerated by user 112 of hearing instrument(s) 102, receive and sendrequest data, receive and send messages, and so on. In some examples,communication unit(s) 504 may include wireless transmitters andreceivers that enable computing device 500 to communicate wirelesslywith the other computing devices. For instance, in the example of FIG. 5, communication unit(s) 504 include a radio 506 that enables computingdevice 500 to communicate wirelessly with other computing devices, suchas hearing instruments 102 (FIG. 1 ). Examples of communication unit(s)504 may include network interface cards, Ethernet cards, opticaltransceivers, radio frequency transceivers, or other types of devicesthat are able to send and receive information. Other examples of suchcommunication units may include BLUETOOTH™, 3G, 4G, 5G, and WI-FI™radios, Universal Serial Bus (USB) interfaces, etc. Computing device 500may use communication unit(s) 504 to communicate with one or morehearing instruments (e.g., hearing instrument 102 (FIG. 1 , FIG. 4 )).Additionally, computing device 500 may use communication unit(s) 504 tocommunicate with one or more other remote devices.

Output device(s) 510 may generate output. Examples of output includetactile, audio, and video output. Output device(s) 510 may includepresence-sensitive screens, sound cards, video graphics adapter cards,speakers, liquid crystal displays (LCD), or other types of devices forgenerating output.

Processor(s) 502 may read instructions from storage device(s) 516 andmay execute instructions stored by storage device(s) 516. Execution ofthe instructions by processor(s) 502 may configure or cause computingdevice 500 to provide at least some of the functionality ascribed inthis disclosure to computing device 500. As shown in the example of FIG.5 , storage device(s) 516 include computer-readable instructionsassociated with operating system 520, application modules 522A-522N(collectively, “application modules 522”), and a companion application524. Additionally, in the example of FIG. 5 , storage device(s) 516 maystore health-related data 526.

Execution of instructions associated with operating system 520 may causecomputing device 500 to perform various functions to manage hardwareresources of computing device 500 and to provide various common servicesfor other computer programs. Execution of instructions associated withapplication modules 522 may cause computing device 500 to provide one ormore of various applications (e.g., “apps,” operating systemapplications, etc.). Application modules 522 may provide particularapplications, such as text messaging (e.g., SMS) applications, instantmessaging applications, email applications, social media applications,text composition applications, and so on.

Execution of instructions associated with companion application 524 byprocessor(s) 502 may cause computing device 500 to perform one or moreof various functions. For example, execution of instructions associatedwith companion application 524 may cause computing device 500 toconfigure communication unit(s) 504 to receive data from hearinginstruments 102 and use the received data to present health-related datato a user, such as user 104 or a third-party user. In some examples,companion application 524 is an instance of a web application or serverapplication. In some examples, such as examples where computing device500 is a mobile device or other type of computing device, companionapplication 524 may be a native application.

In some examples of this disclosure, processor(s) 502 are configured toobtain, one or more signals generated by one or more sensors (e.g.,EMREO sensor(s) 434 and, in some examples, one or more additionalsensors 436) of one or more hearing instruments (e.g., one or more ofhearing instruments 102) that are located within one or more ear canalsof a user (e.g., user 104) of the one or more hearing instruments. Inother words, processor(s) 502 may obtain EMREO-related measurements fromone or more EMREO sensors of a hearing instrument. Processor(s) 502 mayperform one or more actions based on the EMREO-related measurements,such as when the EMREO-related measurements are being indicative of anoccurrence of EMREOs of one or more eardrums of the user of the one ormore hearing instruments. In some examples, rather than receivingEMREO-related measurements directly, processor(s) 502 may receive datagenerated based on the EMREO-related measurements and perform one ormore actions based on the received data. For instance, processor(s) 502may perform actions as describe in any of the examples providedelsewhere in this disclosure.

FIG. 6 is a flowchart illustrating an example operation 600 of thisdisclosure. Other examples of this disclosure may include more, fewer,or different actions. In the example of FIG. 6 , a set of one or moreprocessors (i.e., processing circuits) may obtain EMREO-relatedmeasurements from one or more EMREO sensors (e.g., EMREO sensors 106A orEMREO sensors 106B) of a hearing instrument (e.g., hearing instrument102A or hearing instrument 102B) (600). The EMREO sensors may be locatedin an ear canal of a user (e.g., user 104) of the hearing instrument andare configured to detect environmental signals of EMREOs of an eardrumof the user of the hearing instrument and generate the EMREO-relatedmeasurements based on the detected environmental signals. For instance,in examples where the EMREO sensors include microphones, the microphonesmay generate signals based on changes in air pressure caused by EMREOs;in examples where the EMREO sensors include a VCSEL and a photodetector,the photodetector may generate EMREO-related measurements that indicatea position of an eardrum; and so on as described elsewhere in thisdisclosure. The set of processors may include one or more processors ofhearing instruments 102 and/or one or more processors of computingsystem 108. Thus, in some examples, the hearing instrument includes eachof the processors. In other examples, the hearing instrument includesone or more of processors and one or more other devices (e.g., a deviceof computing system 108, another hearing instrument, etc.) include oneor more of the processors.

Furthermore, in the example of FIG. 6 , the one or more processors mayperform an action based on the EMREO-related measurements (602). Forinstance, example actions may include any of the actions describedelsewhere in this disclosure with respect to the various use cases, orcombinations thereof.

In this disclosure, ordinal terms such as “first,” “second,” “third,”and so on, are not necessarily indicators of positions within an order,but rather may be used to distinguish different instances of the samething. Examples provided in this disclosure may be used together,separately, or in various combinations. Furthermore, with respect toexamples that involve personal data regarding a user, it may be requiredthat such personal data only be used with the permission of the user.

It is to be recognized that depending on the example, certain acts orevents of any of the techniques described herein can be performed in adifferent sequence, may be added, merged, or left out altogether (e.g.,not all described acts or events are necessary for the practice of thetechniques). Moreover, in certain examples, acts or events may beperformed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors, rather than sequentially.

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored on or transmitted over, as oneor more instructions or code, a computer-readable medium and executed bya hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media, or communication media including any mediumthat facilitates transfer of a computer program from one place toanother, e.g., according to a communication protocol. In this manner,computer-readable media generally may correspond to (1) tangiblecomputer-readable storage media which is non-transitory or (2) acommunication medium such as a signal or carrier wave. Data storagemedia may be any available media that can be accessed by one or morecomputers or one or more processing circuits to retrieve instructions,code and/or data structures for implementation of the techniquesdescribed in this disclosure. A computer program product may include acomputer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, cache memory, or any other medium that can be used to storedesired program code in the form of instructions or data structures andthat can be accessed by a computer. Also, any connection is properlytermed a computer-readable medium. For example, if instructions aretransmitted from a website, server, or other remote source using acoaxial cable, fiber optic cable, twisted pair, digital subscriber line(DSL), or wireless technologies such as infrared, radio, and microwave,then the coaxial cable, fiber optic cable, twisted pair, DSL, orwireless technologies such as infrared, radio, and microwave areincluded in the definition of medium. It should be understood, however,that computer-readable storage media and data storage media do notinclude connections, carrier waves, signals, or other transient media,but are instead directed to non-transient, tangible storage media. Diskand disc, as used herein, includes compact disc (CD), laser disc,optical disc, digital versatile disc (DVD), floppy disk and Blu-raydisc, where disks usually reproduce data magnetically, while discsreproduce data optically with lasers. Combinations of the above shouldalso be included within the scope of computer-readable media.

Functionality described in this disclosure may be performed by fixedfunction and/or programmable processing circuitry. For instance,instructions may be executed by fixed function and/or programmableprocessing circuitry. Such processing circuitry may include one or moreprocessors, such as one or more digital signal processors (DSPs),general purpose microprocessors, application specific integratedcircuits (ASICs), field programmable logic arrays (FPGAs), or otherequivalent integrated or discrete logic circuitry. Accordingly, the term“processor,” as used herein may refer to any of the foregoing structureor any other structure suitable for implementation of the techniquesdescribed herein. In addition, in some aspects, the functionalitydescribed herein may be provided within dedicated hardware and/orsoftware modules. Also, the techniques could be fully implemented in oneor more circuits or logic elements. Processing circuits may be coupledto other components in various ways. For example, a processing circuitmay be coupled to other components via an internal device interconnect,a wired or wireless network connection, or another communication medium.

The techniques of this disclosure may be implemented in a wide varietyof devices or apparatuses, including a wireless handset, an integratedcircuit (IC) or a set of ICs (e.g., a chip set). Various components,modules, or units are described in this disclosure to emphasizefunctional aspects of devices configured to perform the disclosedtechniques, but do not necessarily require realization by differenthardware units. Rather, as described above, various units may becombined in a hardware unit or provided by a collection ofinteroperative hardware units, including one or more processors asdescribed above, in conjunction with suitable software and/or firmware.

Various examples have been described. These and other examples arewithin the scope of the following claims.

What is claimed is:
 1. A method comprising: obtaining, by a set of oneor more processing circuits, eye movement-related eardrum oscillation(EMREO)-related measurements from one or more EMREO sensors of a hearinginstrument, wherein the EMREO sensors are located in an ear canal of auser of the hearing instrument and are configured to detectenvironmental signals of EMREOs of an eardrum of the user of the hearinginstrument and generate the EMREO-related measurements based on thedetected environmental signals; determining, by the one or moreprocessing circuits, based on the EMREO-related measurements, potentialmodifications to left-ear and right-ear audio signals; and applying, bythe one or more processing circuits, the potential modifications to theleft-ear and right-ear audio signals in response to determining amovement of a head of the user.
 2. The method of claim 1, whereindetermining the potential modifications to the left-ear and right-earaudio signals comprises determining a delay between the left-ear andright-ear audio signals.
 3. The method of claim 1, wherein determiningthe potential modifications to the left-ear and right-ear audio signalscomprises determining a relative intensity of the left-ear and right-earaudio signals.
 4. The method of claim 1, wherein applying the potentialmodifications to the left-ear and right-ear audio signals comprisesapplying a head-related transform function.
 5. The method of claim 1,wherein applying the potential modifications to the left-ear andright-ear audio signals comprise steering or selecting a microphonebeamformer.
 6. The method of claim 1, wherein the one or more EMREOsensors include a microphone positioned at a medial tip of the hearinginstrument and configured to detect changes in air pressure within theear canals caused by EMREOs.
 7. The method of claim 1, wherein at leastone of: the EMREO sensors include a vertical-cavity surface-emittinglaser (VCSEL) positioned to shine a coherent beam onto the eardrum; andobtaining the EMREO-related measurements comprises applying, by the oneor more processing circuits, optical feedback interferometry based onreflected light of the coherent beam to determine a position of theeardrum, the EMREO sensors include a time of flight (ToF) sensorconfigured to emit infrared light toward the eardrum and configured todetermine a distance to the eardrum based on a travel time of theinfrared light to the eardrum and back to the ToF sensor, and obtainingthe EMREO-related measurements comprises determining, by the one or moreprocessing circuits, the position of the eardrum based on the traveltime, the EMREO sensors include a structured light sensor configured toemit structured light toward the eardrum, and obtaining theEMREO-related measurements comprises determining, by the one or moreprocessing circuits, the position of the eardrum based on a pattern oflight detected by the structured light sensor, the EMREO sensors includea vibration sensor in contact with skin of the ear canal, and obtainingthe EMREO-related measurements comprises obtaining, by the one or moreprocessing circuits, measurements of surface waves in the skin of theear canal caused by the EMREOs of the eardrum, or the EMREO sensorsinclude a vibration sensor attached to the eardrum or an ossicular chainof the user, and obtaining the EMREO-related measurements comprisesobtaining, by the one or more processing circuits, measurements ofvibrations from the vibration sensor.
 8. The method of claim 1, whereinthe hearing instrument is a left hearing instrument of the user, and themethod further comprises: receiving the left-ear audio signal from amicrophone of the left hearing instrument; and receiving the right-earaudio signal from a microphone of a right hearing instrument of theuser.
 9. A system comprising: one or more eye movement-related eardrumoscillation (EMREO) sensors located in an ear canal of a user of ahearing instrument, wherein the EMREO sensors are configured to detectenvironmental signals of EMREOs of an eardrum of the user and generatethe EMREO-related measurements based on the detected environmentalsignals; and one or more processing circuits configured to: determine,based on the EMREO-related measurements, potential modifications toleft-ear and right-ear audio signals; and apply the potentialmodifications to the left-ear and right-ear audio signals in response todetermining a movement of a head of the user.
 10. The system of claim 9,wherein the one or more processing circuits are configured to, as partof determining the potential modifications to the left-ear and right-earaudio signals, determine a delay between the left-ear and right-earaudio signals.
 11. The system of claim 9, wherein the one or moreprocessing circuits are configured to, as part of determining thepotential modifications to the left-ear and right-ear audio signals,determine a relative intensity of the left-ear and right-ear audiosignals.
 12. The system of claim 9, wherein the one or more processingcircuits are configured to, as part of applying the potentialmodifications to the left-ear and right-ear audio signals, apply ahead-related transform function.
 13. The system of claim 9, wherein theone or more processing circuits are configured to, as part of applyingthe potential modifications to the left-ear and right-ear audio signals,steer or select a microphone beamformer.
 14. The system of claim 9,wherein the system comprises the hearing instrument and the hearinginstrument comprises at least one of the EMREO sensors or the one ormore processing circuits.
 15. The system of claim 9, wherein: the systemcomprises a computing system and the hearing instrument, and thecomputing system comprises the one or more processing circuits.
 16. Thesystem of claim 9, wherein the one or more EMREO sensors include amicrophone positioned at a medial tip of the hearing instrumentconfigured to detect changes in air pressure within the ear canal causedby EMREOs.
 17. The system of claim 9, wherein at least one of: the EMREOsensors include a vertical-cavity surface-emitting laser (VCSEL)positioned to shine a coherent beam onto the eardrum, and the one ormore processing circuits are configured such that, as part of obtainingthe EMREO-related measurements, the one or more processing circuitsapply optical feedback interferometry based on reflected light of thecoherent beam to determine a position of the eardrum, the EMREO sensorsinclude a time of flight (ToF) sensor configured to emit infrared lighttoward the eardrum and configured to determine a distance to the eardrumbased on a travel time of the infrared light to the eardrum and back tothe ToF sensor, and the one or more processing circuits are configuredsuch that, as part of obtaining the EMREO-related measurements, the oneor more processing circuits determine the position of the eardrum basedon the travel time, the EMREO sensors include a structured light sensorconfigured to emit structured light toward the eardrum, and the one ormore processing circuits are configured such that, as part of obtainingthe EMREO-related measurements, the one or more processing circuitsdetermine the position of the eardrum based on a pattern of lightdetected by the structured light sensor, the EMREO sensors include avibration sensor in contact with skin of the ear canal, the one or moreprocessing circuits are configured such that, as part of obtaining theEMREO-related measurements, the one or more processing circuits obtainmeasurements of surface waves in the skin of the ear canal caused by theEMREOs of the eardrum, the EMREO sensors include a vibration sensorattached to the eardrum or an ossicular chain of the user, and the oneor more processing circuits are configured such that, as part ofobtaining the EMREO-related measurements, the one or more processingcircuits obtain measurements of vibrations from the vibration sensor.18. The system of claim 9, wherein: the hearing instrument is a lefthearing instrument of the user, and the one or more processing circuitsare further configured to: receive the left-ear audio signal from amicrophone of the left hearing instrument; and receive the right-earaudio signal from a microphone of a right hearing instrument of theuser.